Stochastic Composition Optimization of Functions Without Lipschitz Continuous Gradient

In this paper, we study stochastic optimization of two-level composition functions without Lipschitz continuous gradient. The smoothness property is generalized by the notion relative which provokes Bregman gradient method. We propose three algorithms for possible relatively smooth compositional scenarios and provide their sample complexities to achieve an $$\epsilon $$ -approximate stationary point. For composition, first algorithm requires $$\mathcal {O}(\epsilon ^{-2})$$ calls oracles inner function value as well outer When both are smooth, second ^{-3})$$ oracle gradients oracles. further improve variance reduction setting where just smooth. resulting ^{-5/2})$$ oracle, ^{-3/2})$$ gradient, Finally, numerically evaluate performance these over two different examples.